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5 AN AUGMENTED REALITY SYSTEM AND METHOD 

FIELD OF THE INVENTION 

The present invention relates to an augmented reality system 
10 comprising: a camera for capturing an image, the camera being 
movably located at a local site, a registering unit generating gra- 
phics and registering the generated graphics to the image from 
the camera, to provide a composite augmented reality image, a 
display device located at a remote site, physically separated 
15 from the local site, for displaying a view comprising the compos- 
ite augmented reality image, and a communication link, for com- 
munication of information between the local and the remote site. 

The present invention also relates to a method for remotely dis- 
20 playing an augmented reality view comprising graphical informa- 
tion overlaid an image captured at a local location. 

The system and the method according to the invention is suit- 
able for performing remote tasks and processes, where the op- 
25 erator is physically separated from the location where the tasks 
or processes are preformed. The invention is, for example, use- 
ful in connection with remote robot programming, remote main- 
tenance, inspection as well as remote collaboration. 

30 The present invention also relates to a use of a system accord- 
ing to the invention for remote programming of an industrial ro- 
bot. 
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PRIOR ART 

Augmented reality (AR) is a well-known method of overlaying 
real world representations with computer-generated information. 

5 Ideally, for vision based augmented reality, the user will not be 
able to recognize the difference between the real and the com- 
puter generated graphics, and thereby getting an improved per- 
ception of the real world environment. Today, augmented reality 
techniques are used in a number of applications. New areas. 

10 where augmented reality Is used is for instance within media 
(weather reporting), medicine (visualization of internal organs), 
for collaborative environments (virtual meeting room), and in the 
process industry for maintenance and service tasks. 

15 With traditional vision based augmented reality systems, compu- 
ter generated information is added to a real, local scene. Visual- 
izing a remote environment with overlaid computer generated 
graphics is hereinafter referred to as remote augmented reality. 
Remote augmented reality is useful for tasks, where an expert is 

20 located at a distance from where a physical task is to be exe- 
cuted, such as inspection in hazardous environments, and ser- 
vice and support tasks. 

Collaboration between users in an augmented reality system has 
been addressed in the literature. This can be done by introduc- 
ing images of several remote users as overlaid graphics in a 
real environment. In such a system, meeting participants can 
visualize each other by looking at images of the meeting mem- 
bers rendered in an augmented reality. Further, the remote us- 
ers may communicate by means of virtual information presented 
in the same augmented reality environment. However, it is not 
possible for the remote users not get different views of the local 
scene, seen from different viewpoints. 

35 From the U.S. Patent Application with publication number U.S. 
2002-0010734 A1 it is known to link augmented reality images to 
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a computer network. The document describes an internet work- 
ed reality system comprising one or more local stations, which 
may be AR or non-AR, but at least one of the local stations must 
be AR, and one or more remote stations, which may be AR or 
5 non-AR. The local and the remote stations are networked to- 
gether. The remote stations can provide resources not available 
at the local AR station, such as data bases, high performance 
computing, and methods by which, a person at the remote sta- 
tion can interact with a person at the local AR station. 

10 

In one embodiment disclosed, a trainee is located at the local 
AR station, while an instructor, located at the remote station, 
monitors and controls the training. In a maintenance embodi- 
ment, the operator performs tasks at the local AR station, while 

15 information and assistance are located at the remote AR station. 
The local AR station comprises a head mounted display for use 
by a person at the local station, a camera arranged in a fixed 
relation to the head mounted display, a computer comprising 
software for generating graphical representations, a video mixer 

20 mixing the generated graphical representation with the image to 
achieve a composed augmented reality image. 

The head mounted display is adapted for showing a view com- 
prising the composed augmented reality image. The view shown 
on the head mounted display depends on the orientation and 
position of the head of the user in the local AR station. Informa- 
tion about the composed augmented reality is transferred to the 
remote station through a network connecting the local and the 
remote station. The remote station comprises a display device 
adapted for displaying the composed augmented reality trans- 
ferred thereto. A person located In the remote station will be 
able to view exactly the same view as a person wearing the 
head mounted display in the local station. 




35 U.S. Patent No. 5745387, shows an augmented reality mainte- 
nance system for operating in a hazardous environment, com- 
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prising an environment modeler, which creates a computer mo- 
del of the environment. An environment renderer creates a plu- 
rality of images, each corresponding to a viewing location and 
orientation. A remotely operated manipulator arm is attached at 

5 a fixed end to a stationary structure and is provided with a utility 
package, such as a video camera attached to a distal end of the 
manipulator arm. Actuators move the manipulator arm to desired 
locations within the environment. The position and orientation of 
the manipulator arm are determined by a unit adapted for sens- 

10 ing position and attitude. This information is passed to a mani- 
pulator arm renderer, which creates an image from a pre-stored 
model of the manipulator arm, viewed from several different 
viewpoints. A viewpoint for rendering a computed view of the en- 
vironment is determined by the operator, either by inputting co- 

15 ordinates or by selecting from a pre-established list of viewpoint 
coordinates and provide it to the environment renderer, which 
generates an image corresponding to the shapes defined by the 
environment geometry when viewed from the selected viewpoint. 

The resulting viewpoint, formed by combining the position and 
orientation of the distal end of the manipulator arm and the off- 
set displacements of the utility package, will allow an environ- 
ment rendered to produce images corresponding to views of the 
environment as seen from the viewpoint of the utility package. 
Similarly, position and orientation of the distal end of the ma- 
nipulator arm is provided to a manipulator arm renderer. The 
manipulator arm renderer generates an image of the manipulator 
arm as seen from the same viewpoint used by the environment 
renderer. A video mixer superimposes an image of the manipu- 
lator arm and the image of the environment and displays the su- 
perimposed images on a monitor, thereby allowing an operator 
to visualize a position of the manipulator arm relative to its envi- 
ronment. 




35 



However, none of above described systems enables a remote 
operator to see the local scene from different viewpoints. 



30/12/02 



11:57 



RBB flB PfiTENT 46 21 1811*36 -» 086660286 

46 21 181386 



NR. 905 007 



5 



The invention is particularly suitable in connection with remote 
control of an industrial robot. Generation of high quality robot 
programs, e.g. for painting, grinding, and polishing, requires 

5 skilled programmers with extended process experience. This is 
due to the nature of todays programming systems, where proc- 
ess specific knowledge is important in order to meet quality re- 
quirements. Building up and keeping this competence within the 
end customer's organization is both time consuming and expen- 

10 sive. Cost savings can be accomplished if experts are located 
remotely and thereby can support or perform robot programming 
to several clients. 

OBJECTS AND SUMMARY OF THE INVENTION 

The object of the present invention is to provide an improved 
augmented reality system, which allows a remote operator to 
perform remote tasks or processes in an augmented reality that 
is displayed to the operator in such a way that he experiences 
20 that he is actually on site. 

According to one aspect of the invention, the object is achieved 
by the initially defined system, characterized in that the system 
further comprises a specifying unit, for specification of a posi- 

25 tion and an orientation in the remote site, the camera is ar- 
ranged such that its position and orientation is dependent on the 
specified position and orientation, and the registering unit is 
adapted for registering the generated graphics to the image in 
dependence of the specified position and orientation. In this 

30 context, a local site refers to where the process or task is to be 
executed and a remote site refers to where the operator per- 
forms the process or task. 

The system according to the invention allows a remote operator 
35 to determine which viewpoint to be displayed. The operator spe- 
cifies a position and an orientation corresponding to the desired 
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viewpoint and the position and orientation of the camera is ad- 
justed based on the specified position and orientation and the 
graphics are registered to the image based on the specified po- 
sition and orientation. The system according to the invention al- 

5 lows a remote operator to visualize the environment of a local 
site and overlaid computer generated graphics from different 
viewpoints and thus enables the remote operator to be virtually 
present at the local site. An advantage gained by the invention 
is that the remote operators may see the local scene from dif- 

10 ferent angels and get an improved perception of the environ- 
ment by means of overlaid virtual graphics. There is no need for 
an expert to travel to the local site for service and support. 

According to an embodiment of the invention, said specifying 
15 unit comprises a tracking unit adapted for determining the posi- 
tion and orientation of a movable device located in the remote 
site, in relation to a fixed pre-specified remote coordinate sys- 
tem. Preferably, the movable device is attached to or carried by 
the operator and thus the position and orientation of the opera- 
20 tor determine the view displayed. The tracking unit is either a 
separate unit or a part of the movable device. The tracking unit 
could be any known type of sensor or technique used for deter- 
mining position and orientation. Preferably, said movable device 
is the display device. The display device is preferably a hand- 
25 held display device or a head-mounted display device. Thus, the 
viewpoint displayed depends on the position and orientation of 
the display device. If the operator carries the display device, the 
actual movements of the operator control the position and orien- 
tation of the camera and accordingly the viewpoint displayed. 

30 

According to a preferred embodiment of the invention, the sys- 
tem further comprises a robot located at the local site, the cam- 
era is mounted on the robot and the robot is arranged in such a 
manner that a movement of the robot, in relation to a fixed pre- 
35 specified local coordinate system depends on the specified posi- 
tion and orientation. Preferably, the movements of the robot de- 
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pend of the movements of the movable device. When the user 
wears the movable device, the robot is moved in accordance 
with the operator's movements. Whit a robot is meant an indus- 
trial robot. A robot comprises a manipulator and a control sys- 
5 tern controlling the movements of the manipulator. 

According to a further embodiment of the Invention, the system 
comprises a graphical generator, for generation of a graphical 
representation, and the registering unit is adapted for generat- 

10 ing graphics based on the graphical representation. It is advan- 
tageous to firstly generate a graphical representation and then 
generate graphics based on the graphical representation, since 
the graphical representation can easily be transferred between 
the remote and the local site without any high demand on the 

15 bandwidth. 

According to a further embodiment of the invention, the system 
comprises operator input means located at the remote site, pro- 
vided for feeding data related to graphics to be displayed, to the 

20 system. By the provision of an operator input means for feeding 
data related to the graphics to be displayed, the operator is able 
to add virtual information to the virtual environment, e.g. annota- 
tions or graphical illustrations. In hazardous or unreachable en- 
vironments, it may be necessary to introduce robots in order to 

25 perform inspection of certain properties in a process or device. 
This may be the case in environments with nuclear radiation, 
deep see locations, extreme temperature situations, etc. With a 
remote expert performing this inspection, the remote augmented 
system will be able to provide the remote operator with the re- 

30 quired images of the object to be inspected. This embodiment of 
the invention provides the remote operator with possibility to 
add virtual information to the real environment, providing the 
operator with enhanced perception of the local scene. For in- 
stance, at remote robot programming, the virtual information 

35 may comprise the programmed robot path and process related 
information describing the robot processing result. For remote 
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inspection, the virtual information may comprise a temperature 
or radiation map of the real environment, or status information of 
an object or structure. 

5 According to a preferred embodiment of the invention, the sys- 
tem further comprises a pointing device and a tracking unit for 
determining the position of the pointing device and the system is 
adapted for generating a graphical representation of a point be- 
ing presently pointed out by the pointing member based on the 

10 position of the pointing device. Preferably, the pointing device is 
handheld. Such a pointing device can for example be use by an 
operator at the remote site for teaching point a robot path. In a 
further embodiment, the system comprises two pointing devices 
and two tracking units, the second pointing device and second 

15 tracking unit is used by an operator located at the local site. 

The pointing device is particularly useful in connection with 
teaching a robot program when there is no personnel with robot 
programming skills at the local site, where the processing is to 

20 be performed, or if there is no model of a target object to be 
processed. The pointing device allows a process expert, located 
remotely from where the robot is, to perform a certain task on an 
image of the actual target object. By using the pointing device, a 
remote operator can perform the robot programming task by 

25 teaching points in the virtual world and the points taught are 
shown by a graphical representation. The remote operator may 
wear a head-mounted display, showing a graphical representa- 
tion of the real world as seen from the camera with overlaid 
computer generated graphics. The computer-generated graphics 

30 may, for example, represent the pointing device, process related 
information, and the programmed robot path. The pointing de- 
vice is used to specify waypoints of the robot path and the op- 
erator input means is used to specify process related informa- 
tion to the system. 



35 
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According to one embodiment of the invention, the system com- 
prises a second specifying unit, for specifying a position and an 
orientation in the local site, a second registering unit for genera- 
tion of graphics and for registering of the generated graphics to 

5 the real environment or to an image of the environment of the 
local site in dependence of the position and orientation specified 
by the second specifying unit, and a local display device 
adapted for displaying a view comprising the environment of the 
local site and the graphics projected on the environment. A sec- 

10 ond display device at the local site makes it possible for a local 
operator to supervise the remote operator's actions. This em- 
bodiment of the invention allows the remote operator and the 
local operator to simultaneously view the same environment with 
the same overlaid computer generated graphics from their own 

15 viewpoints. By sharing the same augmented environment, the 
local operator may support the remote operator. The viewpoint 
displayed to the remote operator depends on the position and 
orientation specified by him, e.g. the position of his hand or 
head, and the viewpoint of the local operator depends on the 

20 position and orientation specified by him. 

According to one embodiment of the invention, the system com- 
prises a second movable device located at the local site, and 
the second specifying unit comprises a second tracking unit, de- 

25 termining the position and orientation of the second movable 
device. In a preferred embodiment, said second movable device 
is the local display device. The local display device is, for ex- 
ample, a handheld display device or a head-mounted display. 
Thus, collaboration between a remote and a local operator is 

30 made possible by illustrating the actual movements of the opera- 
tors in the real environment with computer generated graphics. 
System users will be able to interact through a shared virtual 
environment and both the remote and the local user are able to 
change viewpoints. 



35 
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According to one embodiment of the invention, the system also 
comprises a second graphical generator, located at the local 
site, for generating a second graphical representation, and the 
local display device is adapted for displaying a view comprising 
5 the second graphical representation. Thus, it is possible for the 
local operator to generate a graphical representation and add 
graphics to the displayed view. The communication link is 
adapted for communication of the generated graphical represen- 
tations between the local and the remote site. Thus, it is possi- 

10 ble for both the local and the remote operator to modify the 
graphic, but still both operators view the same graphics but from 
different viewpoints. Since only the graphical representation and 
the video signals from the camera are transferred between the 
local and the remote site, instead of the graphics or the entire 

15 composite augmented reality image, the communication band- 
width is reduced. 

According to one embodiment of the invention, the system com- 
prises a second camera for capturing an image, the camera be- 

20 ing arranged in a fixed relation to the second movable device, 
and a second registering unit, registering the generated graphi- 
cal representation to the image from the second camera, to pro- 
vide a composed augmented reality image and that the local dis- 
play device is adapted for displaying a view comprising the 

25 composed augmented reality image. In this embodiment, the 
computer-generated graphics are merged with an image of the 
real world. Alternatively, the computer-generated graphics are 
displayed on see-through glasses worn by the operator. The 
generated graphics are projected on the glasses, so that the op- 

30 erator can see the graphics in relation to the real world and the 
object to be processed. 

According to one embodiment of the invention, the remote dis- 
play device is adapted for displaying a view seen from a first 
35 visual angle that depends on a position and an orientation 
specified at the remote site and the local display device is 
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adapted for displaying the same view as the remote display de- 
vice, seen from a second visual angle that depends on a posi- 
tion and orientation specified at the local site. 

5 According to one embodiment of the invention, the system com- 
prises means for transferring voices between the local and the 
remote site via the communication link. Thus, the remote opera- 
tor and the local operator may communicate by talk to each 
other. Preferably, said communication means comprises micro- 
10 phones or headsets located at the remote and the local site. 
Preferably, the communication link is a network such as the 
Internet. 

According to a further aspect of the invention, the object is 
15 achieved by the initially defined method, comprising specifying a 
position and an orientation in a remote site that is physically 
separated from the local site, positioning and orientating a cam- 
era, located at the local site, according to the specified position 
and orientation, obtaining said image from the camera, generat- 
20 ing graphics, generating a composite augmented reality image 
based on the image, the graphics, and the specified position and 
orientation, and displaying a view comprising the composite 
augmented reality image. 

25 According to another aspect of the invention, the object is 
achieved by a computer program directly loadable into the inter- 
nal memory of a computer, comprising software code portions 
for performing the steps of the method according to the inven- 
tion, when said product is run on a computer. The computer pro- 

30 gram product is provided either on a computer readable medium 
or through a network, such as the Internet. 

According to still another aspect of the invention, the object is 
achieved by a computer-readable medium having a program re- 
35 corded thereon, where the program is to make a computer per- 
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form the steps of the method according to the invention, when 
said program is run on the computer. 

The invention allows tasks or processes, e.g. generation of ro- 
5 bot programs, to be performed from remote locations. Advan- 
tages gained by the invention when used for remote robot pro- 
gramming is: no need for an end-customer to hire and train per- 
sonnel with robot programming competence and the robot pro- 
gramming will become more efficient with a pool of programming 
10 experts in a collaborative environment. 

BRIEF DESCRIPTION OF THE DRAWINGS 

The invention will now be explained more closely by the descrip- 
15 tion of different embodiments thereof and with reference to the 
appended figures. 



20 



Fig. 1 shows an augmented reality system according to an em- 
bodiment of the invention. 

Fig. 2 Is a block diagram of an augmented reality system ac- 
cording to an embodiment of the invention comprising a 
local section and a remote section. 

25 Fig. 3 is a block diagram of a local section of an augmented re- 
ality system according to another embodiment of the in- 
vention. 

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS 
30 OF THE INVENTION 

Fig. 1 illustrates an augmented reality system according to an 
embodiment of the invention. The system comprises one section 
located at a remote site and one section located at a local site. 
35 The local and the remote sections are connected to each other 
via a communication link 1. One or more remote operators 3 are 
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located at the remote site. The remote section comprises a gra- 
phical display device 5, e.g. a head-mounted display device, a 
tracking unit 7 for determining the position and orientation of the 
display device 5 in relation to a fixed remote coordinate system 
5 (xR, yR, zR), a computer 9 containing the necessary software in 
order to generate a graphical representation and an augmented 
reality image, and an communication unit 11 for communication 
with the local site and the computer 9. Preferably, the tracking 
unit 7 has a fixed position and orientation in relation to the dis- 
10 play device 5. The Computer 9 is for instance a wearable com- 
puter. Different types of handhold displays, such as a Personal 
Digital Assistant (PDA) screen could also be used. 

The remote operator 3 also has a pointing device 13, which 

15 makes it possible for the operator to point out positions and ori- 
entations at the remote site. The pointing device 13 comprises a 
tracking unit for determining the position and orientation of the 
pointing device 13. Positions and orientations from the pointing 
device tracking unit and the display tracking unit will be related 

20 to a remote coordinate system (xR, yR, zR), corresponding to a 
local coordinate system (xU, yL, zL) at the local site. In one al- 
ternative, the pointing device comprises a number of interaction 
means, such as pushbuttons, which are adapted for interaction 
between the operator and the system. The interaction means 

25 are, for example, adapted for input of information related to the 
process to the system. Alternatively, the operator feds informa- 
tion to the system via a keyboard or any other known input 
means. The display tracking unit 7 and the pointing device track- 
ing unit is based on any known technology for determining posi- 

30 tion and orientation, such as image recognition, accelerometers, 
inertial sensors, gyros, magnetic trackers, ultrasound, laser 
technology, and a global positioning system (GPS). 

An object 15 to be processed or inspected is positioned at the 
35 local site. The local section of the system comprises one or a 
plurality of robot manipulators 17, and a camera unit 19 mount- 
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ed at one end of a robot arm for capturing stereoscopic or mo- 
noscopic video streams of the object 15. A tracking unit may be 
provided for determining the position and orientation of the ro- 
bot-mounted camera. Alternatively, a system for calculating the 
5 position and orientation of the camera based upon information 
from the robot control system is provided, A robot controller 20 
is connected to the robot manipulator or each of them, making it 
possible to move the robot arm according to a specified position 
and orientation in relation to the fixed local coordinate frame 
10 (xL, yL, zL) in real time. In the following a position and an ori- 
entation is denoted a pose. The local section further comprises 
a communication unit 22 for communicating with the remote 
communication unit 11. 

15 Before the system can be used for the desired purpose, fixed 
remote (xR, yR, zR) and local (xL, yL, zL) coordinate systems 
need to be specified. Poses from the remote site (7, 13) are in 
relation to the remote coordinate system, while poses specified 
at the local site are in relation to the local coordinate system. 

20 Thus, when the remote operator 3 specifies a pose in relation to 
the remote coordinate system, a definition of the local coordi- 
nate system defines the pose of this point in the local environ- 
ment. 

25 All types of cameras can be used, but the demand for accuracy 
of position and orientation determines the type of camera. For 
example, a web camera, a video camera, or a CCD camera 
could be used. In another embodiment, more than one camera 
can be used, for example two cameras could be used arranged 

30 so as to provide stereoscopic images. Any known industrial ro- 
bot having sufficient number of degrees of freedom, preferable 
at least five degrees of freedom, could be used- An industrial 
robot comprises a manipulator having a number of arms mov- 
able relative to each other and a control system controlling the 

35 movements of the arms. 
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In one embodiment of the invention, the system is operated 
without a local operator present. In another embodiment of the 
invention, a local operator is present in order to support the re- 
mote operator. When a local operator 24 is present, the local 
5 section further comprises a graphical display device 26, e.g, a 
head-mounted display, a camera 28 mounted on the display de- 
vice, for capturing a stream of images of the environment and 
the object 15, a tracking unit 27 for determining the position and 
orientation of the display device 26 and the camera 28 in rela- 
10 tion to the local coordinate system, and a computer 32 contain- 
ing the necessary software in order to generate a graphical re- 
presentation and an augmented reality image based upon the 
video stream from the camera 28 and the generated graphical 
representation. 

15 

The camera 28 is mounted in a fixed position in relation to the 
display device 26. Preferably, the display device will be located 
along the camera view axis and at the image plan of the camera. 
In an alternative embodiment, the display device 26 is an optical 

20 see-through display and then, the camera is not needed, since 
the local operator 24 sees the real world environment and the 
object directly through the display device without the camera 
stream. The local section also comprises a hand-held pointing 
device 34 with a tracking unit for determining its position and 

25 orientation in relation to the local coordinate system and means 
for interacting with the system. Preferably, the computer 32 is a 
wearable computer. 

• • • 

• • • 

*\ " The remote operator 3 wears the display device 5 and the move- 

.\ 30 ments of the operator are sensed by means of the tracking unit 
7. The communication between the display unit 5 and the wear- 
able computer 9 is done through a wired or wireless link. The 
:[:*: computer 9 contains the necessary software in order to generate 

:*•]: a graphical representation and an augmented reality image bas- 

35 ed upon the video stream from the camera 19 and the generated 
r'.'l graphical representation. The computer 9 also contains the soft- 
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ware needed to perform a desired task or process, e.g. generat- 
ing a robot program. Finally, the computer 9 will contain a stor- 
age medium in order to save and restore previously saved in- 
formation. 

5 

The communication of information from the pointing device 13 to 
the computer 9 is done through a wired or a wireless link. De- 
pending of the type of tracking system, the pose of the pointing 
device 13 can come from the pointing device itself or from an 

10 external tracking unit. The communication unit 11 contains the 
necessary software and hardware to allow communication from 
the computer 9 to the local communication unit 22. Communica- 
tion from the computer 9 to the communication unit 11 can be 
provided by means of a wired or a wireless communication link. 

15 Communication between the remote 11 and the local 22 com- 
munication units may be done through the Internet or some 
other appropriate communication link. 

The camera 19 is mounted at a distal end of the robot 17 and 
20 transmits images of the environment and the object 15 to the 
remote operator 3. The robot controller 20 receives continuously 
updated poses from the remote display tracking unit 7, so that 
the camera is positioned according to the pose of the remote 
operator 3. The position of the pointing device 34 and the dis- 
25 play device 26 is related to the local coordinate system. The lo- 
cal operator 24 wears the display device 26, which will provide 
the operator with an augmented reality view of the local envi- 
ronment and the object 15. If the display device 26 is of a video 
see-through type, the camera 28 is used for capturing images of 
30 the local environment. If the display device 26 is of an optical 
see-through type, the camera is not needed. 

The local operator 24 uses the pointing device 34 to specify 
poses and interact in the shared virtual environment. The posi- 
35 tion of the pointing device 34 is tracked In relation to the local 
coordinate system. The computer 32 contains the necessary 
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software to produce the local augmented reality view. The com- 
puter 32 generates the process or task related graphics as seen 
by the remote operator. The computer 32 also contains a graph- 
ics unit containing a graphical representation of the virtual world 
5 transferred from the remote section or other local sections via 
the communication link 1, Thus, it is not necessary to transfer 
the graphics in video format between the local and the remote 
site, thereby reducing the communication bandwidth require- 
ments. The computer 32 further comprises a storage device 
10 used to save process related information, e.g. virtual graphical 
representations, or process or task related information provided 
by the remote or local operator. The local site may also contain 
more local operator interacting with the system in the same 
manner. 

15 

Fig. 2 is a block diagram of an augmented reality system ac- 
cording to an embodiment of the invention. The remote section 
of the system, located at the remote site, receives a video 
stream of images from the robot-mounted local camera 19. The 

20 remote operator 3 is able to interact with the system through the 
pointing and interaction device 13. The tracking unit makes it 
possible to trace the pose of the pointing device in 3D, and can 
also be used for specifying the remote coordinate system. In the 
case of robot programming, the pointing device 13 can be used 

25 for specifying waypoints in a robot path and for specifying proc- 
ess related information, while for remote inspection, the pointing 
device may be used for adding virtual information to the object 
or structure to be inspected. 

30 The output from the pointing device 13 and the information fed 
to the system by the operator are transferred to an application 
unit 36. The application unit 36 contains the necessary software 
to perform the desired process or task, e.g. generate a robot 
program or support remote inspections. A graphics unit 37 com- 

35 prises a graphical generator for generating a 3D graphical rep- 
resentation of the visual information that is to be displayed from 
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the application unit 36. The graphics unit 37 comprises specifi- 
cation of a number of 3D graphical elements to be visualized. In 
addition to holding the representation of the graphics from the 
application unit 36, the graphics unit 37 also contains the entire 

5 virtual graphical representation from other remote or local sec- 
tions. The positions of all the graphical elements are specified in 
relation to the remote coordinate system. For the robot pro- 
gramming application, relevant visual information are operator 
specified waypoints, the actual robot path, and task specific in- 

10 formation. Additionally, the pose of the pointing device will be 
rendered so that an operator may get visual indications of what 
other operators are pointing at. The generated graphical repre- 
sentation is transmitted to a registering unit 38. A storage unit 
39 enables saving and loading of application related informa- 

15 tion. 

The display-tracking unit 7 continuously delivers updated data 
regarding the pose of the display device 5. The pose data are 
transmitted to the communication unit 1 1 for further transmission 

20 to the robot controller unit 20. The pose data from the display- 
tracking unit 7 are also transmitted to the registering unit 38. 
The registering unit 38 generates the graphics and based on the 
graphical representation in the graphics unit 37 it registers the 
graphics to the image from the camera unit 19 to provide a com- 

25 posed augmented reality image. The registering unit 38 uses the 
pose data from the display-tracking unit 7 to overlay the com- 
puter generated graphics with the real world images. With the 
registration correctly done, the computer generated graphics will 
be virtually attached to the real world scene. The combined im- 

30 ages are displayed on the display device 5. 

The local communication unit 22 receives continuous informa- 
tion regarding the pose of the remote display device 5 in relation 
to the remote coordinate system and feeds this data to the robot 
35 controller unit 20 steering the robot-mounted camera 19 to the 
same pose in relation to the local coordinate system. The cam- 
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era 19 continuously captures images of the scene, which are 
transmitted back to the communication unit 22 and further to the 
registering unit 38. 

5 Fig. 3 shows another embodiment of the local section at the lo- 
cal site. This embodiment is provided for collaboration between 
a remote and a local operator. The communication unit 22 re- 
ceives the current pose of the remote display device 5, the cur- 
rent pose of the remote pointing device 13, and process specific 

10 information specified by the remote operator, and the graphical 
representations generated in the remote graphics unit 37. The 
received graphical representations are transmitted to a local 
server unit 40. A camera unit 28, which is attached to the dis- 
play device 26, produces local real world images. The pose of 

15 the camera 28 and the display device 26 are determined accord- 
ing to the local coordinate system and is transmitted to a regis- 
tering unit 42 to overlay the real world scene with the computer 
generated graphics generated from a graphics module 44. The 
graphics module 44 receives information from the remote site 

20 regarding the virtual graphical information to be displayed, 
which is a representation of the graphical elements to be ren- 
dered in the shared virtual environment. 

The local operator uses the pointing and interaction device 34 to 
25 interact with the system. The pose of the pointing device will be 
rendered with graphical information in such a way that it is visi- 
ble through the shared virtual environment. Graphical represen- 
tations related to the local operator are transmitted by means of 
the communication unit 22 to the other operators. Information 
30 from the pointing and interaction device 34 is transferred to the 
local graphics module 44 that generates a graphical representa- 
tion of the pointing device. This graphical representation is 
transferred to the local registering unit 42, to the communication 
unit 22, via the server unit 40, and further to the remote graph- 
35 ics unit 37, via the remote communication unit 11. The remote 
graphics module adds the received graphical representation to 
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the graphical representation generated based on the remote in- 
formation. Thus, the remote and the local operator are able to 
see the same view seen from different viewpoints. 

5 The server unit 40 receives data from the remote site through 
the communication unit 22. Further, the server unit 40 holds in- 
formation regarding the local coordinate system. A storage unit 
46, in communication with the server unit 40, is used to store 
the system information, e.g. application related information, gra- 

10 phical representations, and system configuration parameters, 
such as the local coordinate system. One or several local opera- 
tors may be attached to the server unit 40. 

The system is adapted for transmitting voices to/from the remote 
15 operator via the communication link to/from the local operator. 
By equipping both local and remote operators with microphones 
and headsets, it will be possible for the operators to communi- 
cate with each other by means of voice. 

20 The following comprises a description of how the augmented re- 
ality system according to the invention could be used. Either the 
local or the remote operator can initiate a session. Alternatively, 
a session is scheduled In advance. At first, the local coordinate 
system needs to be specified. One way of doing this by a six 

25 point calibration method, where the operator specifies six points 
describing the position and orientation of the coordinate system. 
All pose information provided by the local tracking units will be 
in relation to this local coordinate system. The position and ori- 
entation of the robot or robots in relation to this local coordinate 

30 system also needs to be known. This way, a remotely specified 
point can be mapped to the robot coordinate system. 

A remote operator starts operating the system by wearing the 
display device. The remote operator then needs to specify the 
35 remote coordinate system, which will be corresponding with the 
local coordinate system. The remote coordinate system may be 
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specified in the same manner as the local word coordinate sys- 
tem. All pose information provided by the remote tracking units 
will be in relation to this remote coordinate system. At the local 
site, a target object is positioned at a desired position in relation 
5 to the local coordinate system. 

When the remote operator starts using the system, the move- 
ments of the remote display device will be captures by the dis- 
play tracking unit and transmitted to the local application server 

10 by means of the communication link. The local communication 
server then sends a command to the robot controller to move 
the camera to the pose specified by the remote display-tracking 
unit. Now. the pose of the robot-mounted camera will move to 
the same pose related to the local coordinate system, as the 

15 tracking unit will be in relation to the remote coordinate system. 

The pose of the remote pointing device is tracked in relation to 
the remote coordinate system. The pose information regarding 
the remote pointing device is transferred to the local communi- 

20 cation unit. If a local operator is present, the position of the re- 
mote operator's pointing device will be used to render a com- 
puter generator graphics on the local display device of the local 
operator. Thus, the local operator will be able to visualize the 
movements of the remote operator's pointing device in real time 

25 by means of virtual 3D graphics overlaid the real scene. 

The remote user may now perform the desired process, e.g. ge- 
nerate a robot program. During the execution of a specific task, 
process related information is visualized with computer ge- 

30 nerated 3D graphics on the remote display device. The graphics 
is overlaid the world representation from the local camera and is 
fixed with respect to the remote coordinate system. This makes 
it possible for the remote user to virtually move around the tar- 
get object with a computer-generated graphics attached to the 

35 image of the target object. The movement of the remote opera- 
tor will be captured and transmitted to the local robot-mounted 
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camera in such a way that his camera moves according to the 
remote operator. 

If the remote user performs some sort of off-line task, e.g. gen- 
5 erating a robot path program* the local operator will also be able 
to visualize the process result in the same manner as the re- 
mote operator by means of 3D computer-generated graphics dis- 
played on the local display device. The process graphics will be 
registered and overlaid the real world representation as seen 
10 from the local operator, hence when the local operator moves to 
a new position, the computer-generated process graphics will be 
visualized as fixed with respect to the local coordinate system. 

If the local operator is also equipped with a pointing and interac- 
15 tion device, the position of the local pointing device is visualized 
to the local and the remote operator. The position of the local 
pointing device is tracked in relation to the local coordinate sys- 
tem. Pose information regarding the local pointing device is 
transferred to the remote site via the local and remote communi- 
20 cation units. The pose information of the local operator's point- 
ing device is used to render computer-generated graphics on the 
remote display device and the local display device. Thus, the re- 
mote operator will be able to visualize the movements of the 
pointing device of the local operator. If there are more remote 
25 operators, each remote operator will be able to visualize the 
movements of the pointing devices of the other remote operators 

The software used for implementing the method according to the 
invention is partly based on software known to those skilled in 

30 the art. For example, the positions and orientations of the point- 
ing member may be generated in ARToolKit. The ARToolKit is 
developed by Washington University and the University of Hiro- 
shima and is an open-source software library that enables aug- 
mented reality applications to be built using accurate computer 

35 vision-based tracking techniques. For the application interface, 
the software Open GL may be used. Open GL provides a library 
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of 2D and 3D functions including modeling alterations, color, 
light and shade functions. Microsoft Vision SDK is a library for 
writing programs to perform image manipulation and analyses 
on computers. The augmented reality software includes algo- 
5 rithms for drawing graphics, such as points and lines, transfer- 
ring positions and orientations between different coordinate sys- 
tems, extracting and generating a sequence list of positions and 
orientations, capturing process related information, and drawing 
advanced graphics, such as color-coded points and lines repre- 
10 senting paint strokes with different widths and orientations. 

The present invention is not limited to the embodiments dis- 
closed but may be varied and modified within the scope of the 
following claims. For example in an alternative embodiment, the 

15 remote site is provided with more than one camera, more than 
one display device, more than one specifying unit, more than 
one graphical generator, and more than one registering unit to 
enable two or more remote operators to simultaneously view the 
environment of the local site and the overlaid graphics from their 

20 own view. In another alternative embodiment, the local site is 
also provided with more than one camera, more than one dis- 
play device, more than one specifying unit, more than one 
graphical generator, and more than one registering unit to en- 
able two or more local operators to simultaneously view the en- 

25 vironment of the local site and the overlaid graphics, from differ- 
ent view points. 

In an alternative embodiment, the operator feeds the position 
and the orientation to the system. For instance the operator spe- 
30 cifies the position and orientation by means of one or more joy- 
sticks. In this embodiment the display device is a stationary 
computer screen. 
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CLAIMS 

1. An augmented reality system comprising: 

a camera (19) for capturing an image, the camera being 
5 movably located at a local site, 

a registering unit (38), generating graphics and registering 
the generated graphics to the image from the camera, to provide 
a composite augmented reality image, 

a display device (5) located at a remote site, physically 
10 separated from the local site, for displaying a view comprising 
the composite augmented reality image, and 

a communication link (1), for communication of information 
between the local and the remote site, 

characterized in that the system further comprises a specifying 
15 unit (7), for specification of a position and an orientation in the 
remote site, the camera is arranged such that its position and 
orientation is dependent on the specified position and orienta- 
tion, and the registering unit is adapted for registering the ge- 
nerated graphics to the image in dependence of the specified 
20 position and orientation. 

2. A system according to claim 1. characterized in that said 
specifying unit comprises a tracking unit (7), adapted for deter- 
mining the position and orientation of a movable device (5) lo- 

25 cated at the remote site, the registering unit (38) is adapted for 
registering the generated graphics to the image in dependence 
of the position and orientation of the movable device, and the 
camera (19) is arranged such that its position and orientation 
are dependent on the position and orientation of the movable 

30 device. 

3. A system according to claim 2. characterized in that said 
movable device is the display device (5). 

35 4. A system according to any of the previous claims, charac- 
terized in that it further comprises a robot (17, 20) located at the 
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local site, the camera is mounted on the robot and the robot is 
arranged in such a manner that the movement of the robot de- 
pends on the specified position and orientation. 

5 5. A system according to any of the previous claims, charac- 
terized in that it further comprises a graphical generator (37), for 
generation of a graphical representation, and the registering unit 
(38) is adapted for generating graphics based on the graphical 
representation. 

10 

6. A system according to any of the previous claims, charac- 
terized in that it further comprises operator input means (13). 
located at the remote site, provided for feeding data related to 
the graphics to be displayed to the system, and the system is 

15 adapted for generating the graphics based on said data. 

7. A system according to claim 6, characterized in that said 
operator input means comprises a pointing device (13) and a 
tracking unit for determining the position of the pointing device 

20 and that the system is adapted for generating a graphical repre- 
sentation of a point being presently pointed out by the pointing 
member based on the position of the pointing device. 

8. A system according to any of the previous claims, charac- 
25 terized in that it comprises a second specifying unit (27), for 

specifying a position and an orientation in the local site, a sec- 
ond registering unit (42), generating graphics and registering 
the generated graphics to the real environment or an image of 
the environment of the local site, in dependence of the position 
30 and orientation specified by the second specifying unit (27), and 
a local display device (26) adapted for displaying a view com- 
prising the environment of the local site and said generated 
graphics projected on the environment. 

35 9. A system according to claim 8, characterized in that it com- 
prises a second movable device (26) located at the local site, 
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the second specifying unit comprises a second tracking unit 
(27), for determining the position and the orientation of the sec- 
ond movable device. 

6 10. A system according to claim 11, characterized in that said 
second movable device is the local display device (26). 

11. A system according any of the claims 9-10, characterized in 
that it comprises a second camera (28) for capturing an image, 

10 the camera being arranged in a fix relation to the second mov- 
able device (26), and the second registering unit (42), is 
adapted for registering the generated graphics to the image from 
the second camera (28), to provide a composite augmented re- 
ality image, and that the local display device is adapted for dis- 

15 playing a view comprising the composite augmented reality im- 
age. 

12. A system according to any of the claims 8-11, characterized 
in that the remote display device (5) is adapted for displaying a 

20 view seen from a first visual angle that depends on the position 
and orientation received from the first mentioned specifying unit 
(7) and the local display device (26) is adapted for displaying 
the same view as the remote display device (5) seen from a 
second visual angle that depends on the position and orientation 

25 received from the second specifying unit (27). 

13. A system according to any of the previous claims, charac- 
terized in that it comprises means for transferring voices be- 
tween the remote and the local site via the communication link. 

30 

14. A system according to any of the previous claims, charac- 
terized in that the communication link is a network. 

15. A method for remotely displaying an augmented reality view 
35 comprising graphical information overlaid an image captured at 

a local site, the method comprising: 



30/12/02 11:57 ABB flB PATENT 46 21 181L*36 * 086660286 NR. 905 029 

46 21 181386 

27 



specifying a position and an orientation at a remote site that 
is physically separated from the local site, 

positioning and orientating a camera (19), located at the lo- 
cal site, according to the specified position and orientation, 
5 - obtaining an image from the camera, 

generating graphics, 

generating a composite augmented reality image based on 
the image, the graphics, and the specified position and orienta- 
tion, and 

10 - displaying a view comprising the composite augmented real- 
ity image. 

16. A method according to claim 15, wherein specifying a posi- 
tion and an orientation comprises determining the position and 

15 orientation of a movable device (5) located at the remote site 
and the camera is positioned and orientated according to the 
position and orientation of the movable device (5). 

17. A method according to claim 16, wherein said movable de- 
20 vice is a remote display device (5) and that said view comprising 

the composite augmented reality image is displayed on the re- 
mote display device. 

18. A method according to any of the claims 15-17, comprising 
25 controlling the movements of a robot (17), having the camera 

(19) mounted thereon, according to the position and orientation 
of the movable device (5). 

19. A method according to any of the claims 15-18, comprising 
30 obtaining data related to the graphics to be displayed, and gen- 
erating the graphics based on said data. 

20. A method according to any of the claims 15-19, comprising 
receiving information about the position of a pointing device (13) 

35 and generating graphics representing a point, being presently 
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pointed out by the pointing member, based on the position of the 
pointing device (13). 

21. A method according to any of the claims 15-20 f comprising 
5 specifying a position and an orientation in the local site, and dis- 
playing a second view comprising the environment of the local 
site and the generated graphics projected on the environment in 
dependence of the locally specified position and orientation. 

10 22. A method according to claim 21, wherein specifying a posi- 
tion and an orientation in the local site comprises determining 
the position and orientation of a second movable device (26) lo- 
cated at the local site. 

15 23, A method according to claims 22, wherein the second mov- 
able device is a local display device (26) and that said second 
view, comprising the environment of the local site and the 
graphics, is displayed on the local display device, 

20 24. A method according to claim 22 or 23, comprising capturing 
an image from a second camera (28) being arranged in a fix re- 
lation to the second movable device (26), and registering the 
generated graphics to the image from the second camera (28), 
to provide a composite augmented reality image, and displaying 

25 a view comprising the composite augmented reality image on 
the local display device (26). 

25. A method according to any of the claims 21-24, comprising 
generating second graphics and displaying the second view 

30 comprising the environment of the local site and the second gra- 
phics projected on the environment in dependence of the speci- 
fied position and orientation. 

26. A method according to claim 25, comprising generating a 
35 local graphical representation, generating a remote graphical re- 
presentation, transferring the local and remote graphical repre- 
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sentations between the local and the remote site, generating the 
first mentioned graphics based on the local and the remote gra- 
phical representation, and generating the second graphics bas- 
ed on the local and the remote graphical representation. 

5 

27. A method according to any of the claims 21-26. wherein the 
view displayed in the remote site comprises the environment of 
the local site and the overlaid graphics seen from an visual an- 
gle that depends on the position and orientation specified in the 
10 remote site and the view displayed in the local site comprises 
the environment of the local site and the overlaid graphics seen 
from an visual angle that depends on the position and orienta- 
tion specified in the local site. 

15 28. A computer program product directly loadable into the inter- 
nal memory of a computer, comprising software code portions 
for performing the steps of any of the claims 15-27, when said 
product is run on a computer. 

20 29. A computer readable medium having a program recorded 
thereon, where the program is to make a computer perform the 
steps of any of the claims 15-27, when said program is run on 
the computer. 

25 30. Use of a system according to any of the claims 1-14 for re- 
mote programming of an industrial robot. 
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ABSTRACT 

An augmented reality system comprising: a camera (19) for cap- 
turing an image, the camera being movably located at a local 
site, a registering unit (9). generating graphics and registering 
the generated graphics to the image from the camera, to provide 
a composite augmented reality image, a display dev.ce (5) lo- 
cated at a remote site, physically separated from the local site, 
for displaying a view comprising the composite augmented real- 
ity image, and a communication link (1), for communication of 
information between the local and the remote site, and a speci- 
fying unit (7). for specification of a position and an orientation in 
the remote site. The registering unit is adapted for registering 
the generated graphical representation to the image in depend- 
ence of the specified position and orientation, and the camera is 
arranged such that its position and orientation is dependent on 
the specified position and orientation. 



(Fig. D 
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